Teaching Meta-Analysis for Systematic Reviewers with Mixed Statistical Training

David

Nickson

Coralie

Peter

Background

Meta-analysis is a frequent consultation topic at Stats Central.

I had an HDR student indicate they were doing a network meta-analysis, in a setting where it was clearly not appropriate. Upon interrogation, the supervisor responded: “So it is pushing the envelop. Isn’t that what research is supposed to do?” Neither of them really understood what network meta-analysis is but the supervisor was very determined to use it.

The belief that a statistics background is not needed to do meta-analysis is prevalent—Even in cases where more advanced methods, such as network meta-analysis, were the best approach. The sentiment that systematic reviews are meta-analyses is also common.

I have found that faculty and more advanced HDR students often have a good handle on the amount of work and complexity required. On the other hand, there are also some not aware that, e.g., RevMan isn’t the ideal tool. When I have a consultation with someone who only knows about RevMan or CMA, I know I have a lot more work to do on my end.

We designed and taught a meta-analysis short-course this year.

Literature and existing resources

Systematic reviews, and meta-analyses if appropriate, serve many critical roles in a variety of disciplines.

There are good resources on how to do a meta-analysis, such as:

Cheung & Vijayakumar (2016)—“A guide to conducting meta-analysis”.
Nakagawa et al. (2023)—“Quantitative evidence synthesis: A practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences”.

However, teaching with these resources as-is were not suitable for the majority of our clients. Especially, with advanced meta-analytical models and those that were field-specific, e.g.,

Hu et al. (2019)—“How to read and interpret the results of a Bayesian network meta-analysis: a short tutorial”.
Glisic et al. (2023)—“A 7-step guideline for qualitative synthesis and meta-analysis of observational studies in health sciences”.

The high heterogeneity in statistical training is not just a coincidence or unique to our clients at Stats Central. Systematic reviews and meta-analyses, usually regarded as being at the top of the hierarchy of evidence, are often published in fields such as medical sciences, social sciences, and ecology: fields where statistics is taken as a degree requirement and is often taught in service-teaching capacity (Bromage et al., 2021; Fawcett, 2017; Mustafa, 1996). In other words, where feasible, ‘everyone always wants to do a meta analysis’! Consequently, the level of statistical training of researchers who conduct systematic reviews is naturally quite varied.

Now, there are existing, statistically-sound guides and tutorials on how to conduct meta-analysis such as Nakagawa et al., 2023, and Cheung & Vijayakumar, 2016; however, these resources oft assume that the reader is already familiar with statistics and statistical notation. Emphasise that they are bite sized compared to the cochrane handbook!

As for other resources, there are that focus on the production of meta-analysis outputs, like Zhang et al., 2022 and Balduzzi et al., 2019, which use curated systematic review datasets and can be thus be lacking when actually attempting a meta analysis, given the importance of the data extraction stage.

There are also resources for advanced meta-analytical topics, such as Bayesian methods (Hu et al., 2019). These obviously will be hard to understand without an understanding of basic meta analysis first.

Finally, there are resources tailored to specific fields, such as clinical health (Glisic et al., 2023) or ecology and evolution (Osenberg et al., 1999), which may limit their relevance or accessibility for students from diverse backgrounds. All in all, to our knowledge there isn’t really a resource that comfortably guides the user through the entire process of meta analysis regardless of their background.

What are the pedagogical challenges?

Balance of theory and application of meta-analysis.
How to communicate to learners with noticeable heterogeneity in statistical training.
Strategies to assess learners without formal assessments. Mastery learning (Guskey, 2007) provides a suitable framework?

Course structure ~ Learning outcomes

Effect sizes and sampling variances

Understand the concept of an effect size.
Understand the concept of a sampling variance.

Data extraction guidelines for meta-analysis

Extract and process standard study-level data.
Strategies to extract non-standard study-level data.

The meta-analytical model

Apply fixed and random effects meta-analysis models.
Why and how to control for moderators.

Diagnosing a meta-analytical model

Methods to detect publication bias.
Methods to detect violations of other model assumptions.

Straightforward extensions to the meta-analytical model

Understand and apply meta regression and subgroup analysis.
Other miscellaneous topics in meta analysis.

Introduction to network meta-analysis

Understand and run network meta analysis models.
Diagnose and interpret network meta-analysis results.

Choice of software for the course

RevMan

The go-to choice that is often paired with Covidence.

Point-and-click.
Proprietary.

jamovi (and MAJOR)

Our preferred option until we noticed limitations with MAJOR.

Point-and-click.
Open-source.

R (and RStudio)

Great, but students then need to know R.

Code (and markdown).
Open-source.

Pedagogical insights from 2025

The notation and maths, in the diluted way it was presented (3 main slides over the entire course), was more appreciated than expected.

Pedagogical insights from 2025

The notation and maths, in the diluted way it was presented (3 main slides over the entire course), was more appreciated than expected.
A minority were interested in advanced topics: Multi-level meta analysis models, Bayesian meta analysis.
Data extraction took longer and was more difficult to teach than expected - so many possibilities!
Teaching the concept of a sampling distribution sampling variance via simulation worked well.

Feedback from 2025

Exercises and examples.
Opportunities to work through the exercises with the support of the lecturers.
Discussion of problems in real time.
Very organized step by step instructions to run a meta-analysis.
Practical information about diagnostics and extensions to the base model.

More integrative.
More time to work through the data extraction section.
Explanations about the coding functions.
More practice, especially real scenarios.
Spend more time on non-standard data extraction.

Our study for 2026

Investigate the impact of a meta-analysis short course on student understanding of meta-analytical methods.

Quantify students’ statistical training as we hypothesise that the level of training in systematic reviewers is quite variable.
Quantify if the meta-analysis short course we designed improved students’ understanding of meta-analysis, regardless of their statistical training.

Our study ~ Statistical training

Teaching into other disciplines is a familiar topic in Statistics Education research.
However, the focus of this research is at the undergraduate-level in higher-education contexts.
Hayat et al. (2021) developed a ten question survey to assess the statistical knowledge of health science faculty.

Our study ~ Statistical training questions

Some questions were fine as-is and some questions were not.

Which of the following best describes the primary difference between linear regression and logistic regression?

The dependent variable is categorical in a linear regression while it is continuous in logistic regression.
The dependent variable is continuous in a linear regression while it is categorical in logistic regression.
They serve the same purpose so can be used interchangeably.
I do not understand regression and do not want to guess.

How would you interpret a 95% confidence interval for the true mean of a numeric health outcome?

You can be 5% confident that the interval will not include the true mean.
You can be 95% confident that the interval will include the true mean.
If you draw repeated random samples and calculate a confidence interval for each, you can expect 95% of the intervals to contain the true mean.
I do not understand the expression and do not want to guess.

You can be 95% confident that the sample mean is within the confidence interval.
You can be 95% confident that the true mean is within the confidence interval.
You can be 95% confident that the probability of the true mean being within this confidence interval is 95%.
I do not understand the expression and do not want to guess.

Our study ~ Understanding of meta-analysis

We started with six-point Likert scale questions that will be asked before and after the short-course.

I understand how a meta-analytical model synthesises the same effect size from different studies.

○ Strongly Disagree ↔︎ ○ Disagree ↔︎ ○ Somewhat Disagree ↔︎ ○ Somewhat Agree ↔︎ ○ Agree ↔︎ ○ Strongly Agree

Human ethics rightly pointed out that we should consider test-styled questions instead.

Which model assumes that each study estimates the same true effect size?

Fixed-effect model.
Random-effects model.
Mixed-effects model.
Bayesian hierarchical model.
I am unsure and I prefer not to guess.

Future Directions

Use the feedback and insights to improve the short-course for the 2026 cohort.
Recruit the 2026 cohort for our study and hopefully collect enough data.
Prepare our conference paper at ICOTS 12!